[Ingestion] Update Terraform to deploy ingestion helper as a cloud run service.#1981
[Ingestion] Update Terraform to deploy ingestion helper as a cloud run service.#1981gmechali wants to merge 2 commits intodatacommonsorg:masterfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request migrates the ingestion helper from a Cloud Function to a Cloud Run service and introduces configuration variables for the Spanner project and GCS bucket. Key feedback includes addressing a hardcoded project number in the service account which hinders portability, updating the workflow to reference the new Cloud Run service URI, and avoiding the use of the ':latest' image tag to ensure deterministic deployments.
I am having trouble creating individual review comments. Click here to see my feedback.
import-automation/terraform/main.tf (317-318)
The service account is hardcoded with a specific project number (965988403328), which makes the Terraform configuration non-portable and will cause failures in other GCP projects. Furthermore, using the default compute service account is a security anti-pattern. The Cloud Run service should use the dedicated automation_sa service account, which has already been granted the necessary IAM roles for Spanner and GCS access.
service_account = google_service_account.automation_sa.email
import-automation/terraform/main.tf (351-357)
The spanner_ingestion_workflow needs to be updated to call the new Cloud Run service instead of the now-deleted Cloud Function. The current workflow YAML (as seen in the context) constructs a Cloud Function URL which is no longer valid. You should pass the Cloud Run service URI as an environment variable and update the workflow YAML to use sys.get_env("INGESTION_FUNCTION_URL").
user_env_vars = {
LOCATION = var.region
PROJECT_ID = var.project_id
SPANNER_PROJECT_ID = local.spanner_project_id
SPANNER_INSTANCE_ID = var.spanner_instance_id
SPANNER_DATABASE_ID = var.spanner_database_id
INGESTION_FUNCTION_URL = google_cloud_run_v2_service.ingestion_helper_service.uri
}
import-automation/terraform/main.tf (84)
Using the :latest tag for the container image can lead to non-deterministic deployments. If the image is updated in the registry, new deployments or auto-scaling events might pull a different version than intended. It is recommended to use a specific version tag or a container image digest for production environments.
No description provided.